Remote imaging systems are in use in many applications ranging from convenience store monitoring cameras to sophisticated imaging systems on satellites in space. Such systems typically include an imaging sensor comprised of light-gathering optics, a detector array and support electronics that produce an electronic video signal transmitted to a remote site for human operator viewing and recording.
The imaging sensor is often rotated about one or more axes (usually using a set of gimbals) or translated in space to view an area that is larger than that covered by the sensor's field of view. Often, the sensor's optical system contains a zoom lens or switchable elements that allow the field of view to be changed for greater resolution, or for a larger field of view. In most applications, the images received from the remote sensor are recorded for later viewing and/or processing by observers at remote sites.
Image registration may be defined as a process of spatially aligning two or more images of a scene. This basic operation is often used for remote sensing applications that involve two or more images of a region of interest. Image mosaicking may be defined as a process of smoothly piecing together overlapping images of a scene into a larger image. This operation is used to increase the area of coverage of an image without sacrificing its resolution. An image mosaic is created from a set of overlapping images by registering and resampling all images to the coordinate space of one of the images.
When multiple cameras are positioned close to each other in a camera housing, the distance to the scene, or the region of interest, may be assumed to be the same. The distance and elevation angle to the scene, however, may likely change from one set of images to the next set of images. If this change is not negligible, overlapping images may have geometrical differences, which need to be considered.
Typically, a transformation model representing local geometric differences between overlapping images are used to register the images. The transformation model considers geometrical differences in the overlapping regions between each camera in the housing, and geometrical differences occurring in each camera due to temporal changes in the distance and elevation angles to the scene. Thus, achieving precision registration among multiple cameras is a difficult problem.
The transformation model is typically aided by a host of sensors found in an aircraft, or in an unmanned airborne vehicle (UAV). The sensors may include an inertial measuring unit (IMU) with gyros and accelerometers. These sensors provide independent information about the motion of the cameras disposed in the camera housing (or rig). The information, or metadata, includes the intrinsic parameters of each camera. It also provides the orientation of each camera with respect to a World Coordinate System (WCS) for each frame in a sequence of frames.
Image registration may also be enhanced by various features known to be found within the scene, or region of interest. Such features may include known landmarks. Thus, feature based registration first processes image intensities to identify unique landmarks, or control points in the images. These control points are then used to register the sequence of images.
As these control points move from one image to the next image, the motion may be quantified and used in the temporal characterization of the sequence of images. This motion, as seen by the sequence of images, is known as optical flow. As used herein, optical flow is a two-dimensional velocity field in the image plane induced by relative motion of an imaging sensor (camera) and/or an object(s) in the field-of-view (FOV).
Optical flow, at the pixel level, generally assumes a constancy in the spatial and temporal variations of the image brightness in the FOV. While this may be a valid assumption with an array of electro-optical (EO) cameras housed in an aircraft rig, it generally is not valid with an array of infrared (IR) cameras housed in a similar aircraft rig.
When considering optical flow, it is difficult to match overlapping images taken by a single camera (intra-camera) housed in an aircraft. It is even more difficult to match overlapping images taken by multiple cameras (inter-camera) housed in the same aircraft. The latter difficulty stems from varying image characteristics resulting from pixel variations among different cameras.
Thus, temporal registration of intra-camera images and temporal registration of inter-camera images are a continuing problem, when using multiple EO cameras disposed in an aircraft. The problem is compounded, when using multiple IR cameras in an aircraft. The present invention, as will be explained, deviates from temporal registration used by conventional methods. Instead, the present invention uses a photogrammetric, hierarchical approach in registering images between intra-camera frame sequences and registering images among inter-camera frame sequences. The present invention is described below.